Skip to content

Conversation

@elbeno
Copy link
Contributor

@elbeno elbeno commented Sep 27, 2024

Problem:

  • Quis custodiet ipsos custodies? (Who tests the tests?)

Solution:

  • Mutation testing tests the tests.

Note:

  • To run mutation tests, build the mull_tests target.
  • Mull tests are run as a CI job but this doesn't gate merge_ok.
  • The CI job runs mutations tests on ubuntu-22.04, LLVM 15, mull v0.23.0. This is the current "latest version" taking into consideration the constraints. When we run on ubuntu-24.04, we can use LLVM 17 for mull. And later versions as they are supported.

@elbeno elbeno force-pushed the add-mull branch 3 times, most recently from 5c466d9 to 3dceacc Compare September 27, 2024 16:44
Problem:
- Quis custodiet ipsos custodies? (Who tests the tests?)

Solution:
- Mutation testing tests the tests.

Note:
- To run mutation tests, build the `mull_tests` target.
- Mull tests are run as a CI job but this doesn't gate `merge_ok`.
- The CI job runs mutations tests on ubuntu-22.04, LLVM 15, mull v0.23.0. This
  is the current "latest version" taking into consideration the constraints. When
  we run on ubuntu-24.04, we can use LLVM 17 for mull. And later versions as
  they are supported.
Copy link
Contributor

@mjcaisse-intel mjcaisse-intel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super interesting. 17 is the new 42, but also I can see the value of how it tightens up the tests. This would be a good topic for a FW team sharing.

@mjcaisse-intel mjcaisse-intel merged commit 7976eaf into intel:main Sep 28, 2024
32 checks passed
@elbeno elbeno deleted the add-mull branch October 1, 2024 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants